Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactoring: use tmp path fixture to mock remote and local for transport plugins #6627

Conversation

unkcpz
Copy link
Member

@unkcpz unkcpz commented Nov 21, 2024

These are changes required by #6620, and some changes are overlap with changes in #6626. The changes are focus on following two purposes:

  • Remove the use of deprecated chdir and getcwd methods.
  • Use tmp_path and tmp_path_factory fixtures to mock remote and local folder to test transport plugins.

The change make every test has its own tmp folder therefore can run independent.

@unkcpz unkcpz marked this pull request as draft November 21, 2024 11:52
Copy link

codecov bot commented Nov 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.89%. Comparing base (ef60b66) to head (cd035ad).
Report is 147 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6627      +/-   ##
==========================================
+ Coverage   77.51%   77.89%   +0.38%     
==========================================
  Files         560      567       +7     
  Lines       41444    42180     +736     
==========================================
+ Hits        32120    32850     +730     
- Misses       9324     9330       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@unkcpz unkcpz force-pushed the Remove-the-use-of-deprecated-tranport-methods-in-tests branch 2 times, most recently from aeb255f to 77e0074 Compare November 21, 2024 15:31
@unkcpz unkcpz force-pushed the Remove-the-use-of-deprecated-tranport-methods-in-tests branch from 58b1668 to f19067c Compare November 21, 2024 15:58
@unkcpz unkcpz force-pushed the Remove-the-use-of-deprecated-tranport-methods-in-tests branch from f2c9064 to db72976 Compare November 21, 2024 17:40
@unkcpz unkcpz force-pushed the Remove-the-use-of-deprecated-tranport-methods-in-tests branch from 56527c6 to f9bfea9 Compare November 21, 2024 23:24
@unkcpz
Copy link
Member Author

unkcpz commented Nov 24, 2024

When running tests, I got a lot resource not clean warnings which may indicate some resource leak:

tests/transports/test_all_plugins.py::test_copy[core.local]
  /home/jyu/venv/aiida-core-dev/lib/python3.12/site-packages/psycopg/_connection_base.py:146: ResourceWarning: connection <psycopg.Connection [IDLE] (host=localhost port=46513 user=guest database=01029ab2-f1cd-4c66-a7f1-a405af5e5324) at 0x775bc419b920> was deleted while still open. Please use
 'with' or '.close()' to close the connection
    warn(

This require some investigation.

@unkcpz unkcpz changed the title Remove the use of deprecated tranport methods in tests Refactoring: use tmp path fixture to mock remote and local for transport plugins Nov 24, 2024
@unkcpz unkcpz force-pushed the Remove-the-use-of-deprecated-tranport-methods-in-tests branch 3 times, most recently from a0c5056 to 3c1a702 Compare November 24, 2024 19:04
@unkcpz unkcpz marked this pull request as ready for review November 25, 2024 06:31
Copy link
Contributor

@khsrali khsrali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @unkcpz ,
Thanks a lot! I think this PR is useful.

tests/transports/test_all_plugins.py Show resolved Hide resolved
tests/transports/test_all_plugins.py Show resolved Hide resolved
tests/transports/test_all_plugins.py Show resolved Hide resolved
Comment on lines 455 to 397
def test_put_get_abs_path_file(custom_transport):
def test_put_get_abs_path_file(custom_transport, tmp_path_factory):
"""Test of exception for non existing files and abs path"""
local_dir = os.path.join('/', 'tmp')
remote_dir = local_dir
local_dir = tmp_path_factory.mktemp('local')
remote_dir = tmp_path_factory.mktemp('remote')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to be consistence, can you also define local_tmp_path fixture?
Because now, in some tests only use remote_tmp_path, some other directly uses tmp_path_factory ..
Either that, or I suggest just remove that remote_tmp_path fixture and always to use tmp_path

tests/transports/test_all_plugins.py Show resolved Hide resolved
tests/transports/test_all_plugins.py Show resolved Hide resolved
tests/transports/test_all_plugins.py Show resolved Hide resolved
tests/transports/test_all_plugins.py Outdated Show resolved Hide resolved


def test_exec_pwd(custom_transport):
def test_exec_pwd(custom_transport, remote_tmp_path):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would keep the previous pwd and instead make it exclusive to core.ssh & core.auto_ssh & core.local.
This test would not make sense for plugins without getcwd

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plugins without getcwd

what you mean by plugins without getcwd? I think ssh/auto_ssh/local all has this method implemented, no?

tests/transports/test_all_plugins.py Outdated Show resolved Hide resolved
@unkcpz
Copy link
Member Author

unkcpz commented Nov 25, 2024

This PR require the change in #6639 to not fail the mypy.

Copy link
Member Author

@unkcpz unkcpz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @khsrali for review! I think I address all your comments.

tests/transports/test_all_plugins.py Show resolved Hide resolved
tests/transports/test_all_plugins.py Show resolved Hide resolved
tests/transports/test_all_plugins.py Outdated Show resolved Hide resolved
@unkcpz unkcpz force-pushed the Remove-the-use-of-deprecated-tranport-methods-in-tests branch from cc051b4 to f4de64d Compare November 25, 2024 19:53
@unkcpz unkcpz requested a review from khsrali November 25, 2024 19:55
Copy link
Contributor

@khsrali khsrali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @unkcpz , just a few more adjustments.. then I think would be good to go

tests/transports/test_all_plugins.py Outdated Show resolved Hide resolved
tests/transports/test_all_plugins.py Show resolved Hide resolved
tests/transports/test_all_plugins.py Show resolved Hide resolved
tests/transports/test_all_plugins.py Show resolved Hide resolved
tests/transports/test_all_plugins.py Outdated Show resolved Hide resolved
tests/transports/test_all_plugins.py Show resolved Hide resolved
@unkcpz
Copy link
Member Author

unkcpz commented Nov 28, 2024

Thanks @khsrali, I addressed the comments you mentioned, and improve a test to make sure the getfile is override the retrieved_file download by get method.

@unkcpz unkcpz requested a review from khsrali November 28, 2024 12:16
[pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci
@unkcpz unkcpz force-pushed the Remove-the-use-of-deprecated-tranport-methods-in-tests branch from 8f120ac to cd035ad Compare November 28, 2024 12:17
Copy link
Contributor

@khsrali khsrali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @unkcpz , LGTM!

@unkcpz unkcpz merged commit 197c666 into aiidateam:main Nov 29, 2024
10 checks passed
@unkcpz unkcpz deleted the Remove-the-use-of-deprecated-tranport-methods-in-tests branch November 29, 2024 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants